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(57) Abstract 



A method and apparatus are disclosed 
for classifying materials (16) utilizing a 
computerized touch sensitive screen (9) or 
other computerized pointing device for op- 
erator identification and electronic marking 
of spatial coordinates of materials (16) to be 
extracted. An operator positioned at a com- 
puterized touch sensitive screen (9) views 
electronic images of the mixture of materi- 
als (16) to be sorted as they are conveyed 
past a sensor array (4) which transmits se- 
quences of images of the mixture, either di- 
rectly or through a computer to the touch 
sensitive display screen (9). The operator 
manually "touches" objects displayed on the 
screen (9) to be extracted from the mixture 
thereby registering the spatial coordinates 
of the objects within the computer. The 
computer then tracks the registered objects 
as they are conveyed and directs automated 
devices including mechanical means such as 
air jets (15), robotic arms (18), or other me- 
chanical diverters to extract the registered 
objects. 
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teleoHrated robotic sorting system 



The present invention relates generally to a robotic sorting system, and, 
more particularly to a robotic sorting system suitable for separating recyclable or 
waste material. This invention was made with Government support under Contract 
No. DE-FGO2-95ER82037, having an effective date of Sept. 1, 1995, awarded by 
the United States Department Of Energy. The Government has certain rights in 
the invention. 



BACKGROUND OF THE INVENTION 



In the recycling of waste secondary materials it is very useful to be able to 
separate mixtures of materials into usable fractions. Such separations are 
sometimes performed by fully automated systems which use automated sensors for 
materials identification with subsequent automated extraction of selected materials 
from mixtures such as disclosed in U.S. Patents 5,260,576 and 5,555,984. In 
many instances automated materials handling and sorting systems are combined 
with manual handsorting in a semi-automated process such as that disclosed in 
U.S. Patent 5,411,147. In many other systems, particularly at smaller recycling 
facilities, most of the sorting is done by manual handsorting. 

In a fully automated sorting system identification of materials to be 
extracted from a conveyed stream is performed by automated sensors which are 
specific to identification of certain materials. Advantages to these systems are 
often high speed and the lack of need for manual labor. A disadvantage to these 
systems is that automated sensors are generally limited in their ability to identify a 
wide range of materials and therefore have limited applicability to only selected 
sorting tasks. On the other hand, in manual handsorting the human visual sensory 
system is used to make identifications and advantageously is capable of efficiently 
identifying a wide range of various materials to be sorted. However, also in 
manual handsorting, human hands are used to make the sorting extractions from 
the waste stream with disadvantages of relatively low capacity compared to the 
speed of identifications that a human can perform, and secondly, handsorting 



SUBSTITUTE SHEET (RULE 26) 



BNSDOCID:<WO 9819799A1 I > 



WO 98/19799 



• 




PCT/US97/19680 - 



requires that humans come into contact with a waste strelff Which in general is 
unsanitary and often contains hazardous objects such as broken glass and other 
objects which can easily puncture or cut. 

Present sorting technologies for waste materials generally require the 
presence of an operator on the sorting floor or, in the case of manual handsorting, 
at the sorting line manually extracting recyclable materials from the waste stream. 
This generally requires that the equipment operator or manual handsorters be able- 
bodied. Therefore this type of work has generally not been available to the 
physically handicapped since the physical demands of the work would be beyond 
their capabilities or would place them in a dangerous environment because of their 
physical condition. This situation has lessened job opportunities for the physically 
handicapped. 

Robotic systems have been applied in industry for a number of years 
typically to reduce human labor, reduce human presence in hazardous or 
potentially hazardous situations, and to replace humans in tedious repetitive tasks. 
In U.S. Patent No. 5,299,693 a robotic system is described for extracting 
recyclame materials from a waste stream where the recyclable materials physically 
have a tag coupled to them which provides a non-visual identifying signal which 
can be received by a sensor for identification and with a robotic arm subsequently 
guided into the waste stream to retrieve the tagged item. However, the 
requirement for physical attachment of a signal generating tag to the items to be 
sorted limits the usefulness of the process and would require a massive change in 
practices by the packaging industry to provide such signal markers on packaging 
materials typically found in the waste stream. U.S. Patent No. 5,411,147 
discloses using robotic arms to extract materials from a waste stream although 
details of the robotic systems and interactive functions with humans are not 
provided. U.S. Patent No. 4,942,538 discloses a teleoperated robotic tracker 
which is guided by a human operator using a joystick type hand controller with 
video feedback of robotic arm motion to the operator from a camera mounted on 
the robotic arm. The use of such a system for recyclables sorting would be 
awkward and slow since the human operator would need to provide continual 
guidance to the robotic arm throughout the whole sorting process. There are 
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numerous other robotic^stems with interactive human operation disclosed in the' 
prior art. However, nowhere in the prior an have the inventors found descriptions 
of robotic systems for sorting materials where the identification of selected 
materials to be extracted from a waste stream or other conveyed stream of 
materials is provided by a human operator utilizing a computerized pointing device 
such as a touch screen for electronically registering the spatial coordinates of the 
selected materials with subsequent fully computerized control of a mechanical or 
other robotic system to acquire and extract the selected materials from the 
conveyed stream. 

SUMMARY OF THE INVENTION 

One of the objectives of the present invention is to alleviate one or more of 
the problems identified above. Another objective of the present invention is to 
provide a sorting technology which incorporates the sensing flexibility and sensing 
speed of a human being combined with high speed high capacity mechanical 
material extraction systems to provide a highly flexible high capacity sorting 
system. A second objective is to provide a sorting technology which incorporates 
the sensing flexibility and sensing speed of a human being while insulating the 
human being from contact with the material stream. A third objective is to 
provide a sorting technology which can be operated by the physically handicapped. 
A fourth objective is to provide an automated sorting technology capable of being 
trained by a human operator in order to become fully automated. 

These and other objectives are achieved by providing a method for 
identifying and sorting material objects from a mixture of material objects in 
which the mixture of material objects is conveyed through an inspection zone. The 
mixture of material objects is irradiated with incident electromagnetic radiation in 
the inspection zone. The electromagnetic radiation emanating from the irradiated 
material objects is measured and processed to produce electronic images suitable 
for visual display. Interactive selection of material objects is performed and the 
selected material objects are then separated from the mixture of material objects by 
an automated device. 
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Therefore, the herein disclosed invention overcon^Ttete limitations 
discussed above in automated sorting systems, manual handsorting, and in robotic 
sorting systems by providing for rapid human identification and selection of 
objects to be sorted with subsequent rapid extraction of selected objects by fully 
computerized mechanical means. The invention further provides for use of the 
human capability for rapid and efficient identification and selection of materials to 
be sorted without subjecting the human operator to direct contact with the material 
stream and without requiring that the operator be present on the sorting floor. The 
invention further provides for human training of the sorting system so that the 
sorting system retains the flexibility of the human for identifying selected materials 
for sorting while being fully automated. 

The disclosed invention classifies materials by utilizing a computerized 
touch screen or other computerized pointing device for operator identification and 
electronic marking of spatial coordinates of materials to be extracted from a 
mixture of materials with subsequent computerized position tracking and extraction 
of the marked materials by computer controlled mechanical means. More 
specifically, an operator positioned at a computerized touch screen views 
electronic images of the mixture of materials to be sorted as they are conveyed 
past a sensor array which transmits a sequence of images of the mixture to a touch 
screen either directly or through a computer. The operator views the touch screen 
images and manually "touches'' objects displayed on the screen to be extracted 
from the mixture thereby registering the spatial coordinates of the objects within 
the computer. The computer then tracks the registered objects as they are further 
conveyed and directs mechanical means such as air jets, robotic arms, or other 
mechanical diverters to extract the registered objects from the mixture at an 
appropriate position downstream from the camera position. High speed 
communications between the touch screen, computer, and mechanical sorting 
equipment allows that the touch screen monitor can be located remote from the 
sorting environment such as in an air conditioned office or even at a remote 
location such as across town or in another locality altogether. Therefore there is 
no requirement that the operator be in contact with or even near the material 
stream. 
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The computer is aTsb capable of "learning" the properties of those objects 
being selected for extraction by the operator and capable of taking over the 
selecting process after a sufficient learning process. At that time the system 
becomes a fully automated sorting system for extraction of those types of selected 
objects. At any time the computer can be retrained to select different or additional 
objects by the operator selecting the different or additional objects through the 
touch screen or other computerized pointing device. 



BRIEF DESCRIPTION OF THE DRAWINGS 



The accompanying drawings, which are incorporated in and constitute a 
pan of the specification, illustrate presently preferred embodiments of the 
invention, and, together with the general description given above and the detailed 
description of the preferred embodiments given below, serve to explain the 
principles of the invention. 

Figure 1 is a perspective view of the apparatus according to a first 
preferred embodiment of the invention; 

Figure 2 is a schematic diagram showing material objects diverted by an air 

blast. 

Figure 3 is a bar graph showing the performance of various pointing 
devices. 

Figure 4 is a bar graph showing the number of correct captures per second 
using different pointing devices. 

Figure 5 is a perspective view showing a second embodiment of the 
apparatus according to the invention. 

Figure 6 is a perspective view showing an embodiment of a robotic arm 
according to the invention. 

Figures 7a and 7b are diagrams showing the operation of the robotic arm. 

Figures 8a and 8b are diagrams showing the operation of another 
embodiment of the robotic arm. 

Figure 9 is a block diagram showing the software interfaces for the first 
embodiment of the apparatus of the invention. 

5 
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Figure 10 is a block diagram showing the hardwai^omponents for the 
first embodiment. 

Figure 1 1 is a block diagram showing the software interfaces for the 
second embodiment of the apparatus of the invention. 

Figure 12 is a block diagram showing the hardware components for the 
second embodiment. 



A first preferred embodiment of the invention is shown in Figure 1 . A 
mixture of materials 1 to be sorted are conveyed on conveying surface 2 through 
an inspection zone 3 which is irradiated with electromagnetic radiation from 
radiation sources 5, for instance halogen lamps when sensing is performed in the 
visible light range. Other suitable radiation sources 5 can include, for example, 
Klystron tube (microwave radiation), UV lamp (ultraviolet radiation), IR lamp 
(infrared radiation), X-Ray tube (X-Ray radiation) and a Radio-Nuclide source 
(gamma rays) . Conveying surface 2 may be a belt conveyor, a slide, a vibrating 
pan conveyor, a free fall trajectory, or any other means for conveying materials. 
The material objects comprising the mixture of materials 1 may be conveyed 
singly or in plurality. 

A sensor array 4 (for example, Sony Series 9000 CCD video camera when 
radiation from sources 5 is in the visible light range) is positioned to view the 
inspection zone so to provide data arrays corresponding to measurements of 
electromagnetic radiation emanating from inspection zone 3 and from any 
materials 1 being conveyed through the inspection zone. The electromagnetic 
radiation emanating from materials 1 may be reflected radiation, radiation 
transmitted through materials 1, radiation emitted from materials 1 through 
fluorescence, or any other forms of radiation resulting from interaction of 
materials 1 with the incident radiation from sources 5. Sensor array 4 is shown 
positioned above conveying surface 2 although in practice it may be located at any 
position required to give the desired view. Sensor array 4 is selected to be 
sensitive in the wavelength range of electromagnetic radiation emanating from 
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material objects within the inspection zone 3 when irradiated by sources 5 which 
may be a single source or multiple sources. The geometry of sensor array 4 is 
determined by the application. For instance the sensor array 4 may be a linear 
array of sensors or an area array of sensors. It may physically span the full width 
and/or length of the inspection zone 3 or it may be more compact and use optics 
to scan the width and/or length of the inspection zone such as with a CCD camera. 
Sensor array 4 may be positioned on the same side of the inspection zone 3 as is 
irradiation sources 5 or it may be positioned on the opposite side of inspection 
zone 3 from sources 5 or positioned at any other location with respect to 
irradiation sources 5 and inspection zone 3. The effective wavelength ranges of 
sensor array 4 and sources 5 may each be in any one of the microwave wavelength 
range, the ultraviolet wavelength range, the visible light wavelength range, the 
infrared wavelength range, the x-ray wavelength range, or the gamma ray 
wavelength range or any combination thereof. Sensor array 4 may be fitted with 
special filters which allow only certain wavelengths of electromagnetic radiation to 
reach sensor array 4 for measurement. 

Sensor array 4 data corresponding to electromagnetic radiation 
measurements emanating from the inspection zone 3 and from material objects 1 
within the inspection zone are transmitted from sensor array 4 to computer 7 
and/or touch sensitive screen 9 over transmitting cables 6 or by wireless means. 
Control of sensor array 4 operation may also be provided by computer 7 over 
transmitting cables 6 or by wireless transmission. Sensor array 4 data received by 
computer 7 are processed for analog to digital conversion if not already digital by 
nature of sensor array 4 and microcomputer processed into digitized electronic 
images which are transmitted over cables 8 to touch sensitive screen 9 which is 
capable of electronically registering the coordinates on the screen of a manual 
touch by a human operator 26. Touch screen 9 displays the digitized images 
corresponding to the sensor data from sensor array 4 of the inspection zone 3 and 
the materials to be sorted within the inspection zone 3. Alternatively, the sensor 
array 4 can transmit the images directly to the touch sensitive screen 9. 

Human operator 26 views the electronic images on touch screen 9 and 
manually touches the image of any material object, in this case object 16, which 
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coordinates describing the location on the touch sensitive screen 9 of the touch by 
operator 26 are registered by touch sensitive screen 9 and transmitted over cables 
8 to computer 7. Computer 7 associates the touch screen coordinates of the 
registered touch with corresponding spatial location coordinates on conveying 
surface 2 within inspection zone 3 and further associates any object on conveying 
surface 2 at that location (in this case object 16) with the touch. Computer 7 then 
electronically tracks the motion of selected object 16 as it is further conveyed 
along conveying surface 2 utilizing signals transmitted over cables 28 indicating 
conveyor speed generated by conveyor speed encoder 27. Similarly, the motion 
could be tracked by programming into computer 7 a predetermined speed for 
conveying surface 2 in which case conveyor speed encoder 27 would not be 
needed. As object 16 leaves the inspection zone 3 and drops off the discharge end 
of conveying surface 2 it is diverted from its falling trajectory by air blast 15 as 
shown in Figure 2. Air blast 15 is generated by computer 7 by sending at the 
right time control signals over cables 10 (Figure 1) to the appropriate air solenoid 
valve' 11a within air solenoid array 11 to open air solenoid valve 11a for an 
appropriate length of time to eject object 16. At such time compressed air from 
air reservoir 12 flows through air solenoid 11a and is emitted as an air blast from 
air nozzle 13a within air nozzle array 13, and ejects selected object 16 so that it 
falls over splitter plate 14 and is segregated from non-ejected material objects 17. 
In this way selected objects may be sorted away from a mixture of objects. 

Further details of the present invention are described in the applicants' 
report to the U.S. Department of Energy, Phase II application under Grant 
application number 35853-95-1, titled "Use of Computer Robotics to Reduce 
Human Contact with the Waste Stream and Lower the Costs for Recyclable 
Materials," (hereafter "DOE Report") and in provisional application 60/030,183 
filed November 4, 1996, the disclosures of which are incorporated herein, in their 
entireties. 

Computer 7 may contain a pre-compiled pattern database or identification 
and pattern recognition algorithms which can perform learning of selections by 
operator 26 as the operator makes the selections. Such identification and pattern 
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recognition algorithms riS^be accomplished by computerized neural networks or 
other such pattern recognition computer code. Identification by pattern recognition 
of the objects can be performed by using, for example, the edge enhancement and 
image contour extraction techniques disclosed in the DOE Report at pages 22-29. 
Further details of pattern recognition and its interaction with robotic systems is 
described in the published text titled "Robot Vision," Berthold Klaus Paul Horn, 
MIT Press, Cambridge, Massachusetts (1991), the disclosure of which is 
incorporated herein, in its entirety. 

Learning the recognized object patterns can be performed using known 
neural network or other pattern recognition and learning systems. Neural network 
techniques for identifying and learning patterns are described in, for example, the 
published text "Neural Networks for Pattern Recognition," Christopher M. Bishop, 
Oxford University Press, New York (1995), (hereafter "Bishop"), the disclosure of 
which is incorporated herein, in its entirety. Bishop chapters 3-6 describe neural 
network techniques including single and multiple layer perception as well as 
different error functions that can be used for training neural networks. Bishop, 
chapters 9 and 10 describe techniques for learning and generalization in a neural 
network. 

In this case operator 26 will initially make selections of items to be 
extracted from the mixture of materials such as object 16. As operator 26 makes 
selections the associated electronic images will be processed through the computer 
algorithms with the imaging patterns distinctive to the selected items noted by the 
algorithms. As similar items are repetitively selected by operator 26 the computer 
algorithms associate the distinctive properties of the imaging patterns with objects 
to be selected for extraction and begin to electronically select similar patterns for 
extraction without input from the human operator 26. In this way the 
computerized system learns those objects to be extracted and after sufficient 
learning experience can begin sorting without input from operator 26. 

The choice of using a touch screen 9 for making the selection of objects to 
be extracted from the mixture is a matter of preference. Similar pointing devices 
interfaced to a display screen could be used such as a computer mouse, a track 
ball, a joystick, a touch pad, a light pen, or other such device. The inventors 
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have chosen the touch screen as the preferred pointing de^w based upon their 
intensive studies of some of these various types of devices for sorting applications. 
Figure 3 shows a bar graph of the results of some of these studies. The 
Performance Index assesses the overall performance of a human operator using 
such devices for "pointing and clicking" on certain objects dynamically displayed 
on a computer monitor in order to select these certain objects for sorting in a 
simulation. The Performance Index considers selection accuracy, selection speed, 
and ease of use. Four different cases are presented for each pointing device 
representing various numbers of objects displayed on the monitor screen at a time 
conveyed at different speeds. From the data it is seen that the touch screen 
outperformed all other pointing devices by a wide margin. 

The inventors have found that a human operator can comfortably view 
computerized images of a mixture of materials conveyed past a camera and 
identify and select items to be sorted out of the mixture at a rates up to 2.5 
selections per second using a computerized touch screen to make the identifications 
and selections (see Figure 4). This is two to five times as fast as industry 
established typical manual handsorting rates of one item every one to two seconds. 

Figure 9 shows a block diagram of software interfaces for implementing 
the preferred embodiment of the invention depicted in Figs. 1 and 2. In Fig. 9 the 
sensor array 4 sends data through A/D converter 46 to microprocessor 47. 
Human operator 26 (Fig. 1) touches 48 the touch sensitive screen 9 on the 
displayed image to select object 16 (Fig. 1). This input to touch sensitive screen 9 
is tagged 49 by microprocessor 47 as a selection icon 51 which is scrolled 50 in 
microprocessor 47 host memory and on touch screen 9 as corresponding selected 
object 16 moves through inspection zone 3 (Fig. 1). Microprocessor 47 also 
enters a representation of selection icon 51 into the eject queue 52, assigns to the 
representation of selection icon 51 an appropriate value of air delay 53 to establish 
proper timing for ejection of selected object 16 (Fig. 1) and an appropriate value 
of air on time 54 to effect ejection of selected object 16 as it passes in front of 
ejectors 55 (shown as 11a and 13a in Fig. 2). 
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Fig. 10 shows ^fock diagram of a hardware implementation for the 
preferred embodiment of Figs. 1, 2, and 9. In this case video camera 56 is used 
as sensor array 4 (Figs. 1 and 9) and interfaces to computer 7 (which incorporates 
microprocessor 47) through a frame grabber card 57. Images from video camera 
56 are displayed as digitized or analog electronic images on the touch screen 9. 
User touch input 48 (Fig. 9) is sent back to computer 7 and processed in 
conjunction with the electronic images to eject selected object 16 from a mixture 
of materials 1 (Fig. 1) by activating pneumatic ejectors 55 (shown as 11a and 13a 
in Fig. 2) through ejector controller card 58 at the appropriate time. An 
appropriate CCD video camera for this use is a Sony Model DCX-9000 3CCD 
color progressive scan camera. A representative video card for interfacing with 
the Sony camera is Coreco Model Ultra II with RGB Digitization Module. A 
representative computer is Carlo Gavazzi Model 690 Industrial Computer. An 
appropriate touch screen is PCS Computers Inc. Series G Flat Panel Monitor. An 
appropriate ejector controller card is National Instruments Model PCI-DIO-96. 
Appropriate pneumatic ejectors are comprised of MAC Valves Inc. 6200 air 
solenoid valves coupled to Industrial Spray Products VeeJet Standard nozzles. 

Fig. 5 shows a second preferred embodiment of the apparatus according to 
the invention. A mixture of materials 1 to be sorted are conveyed on conveyor 2 
through an inspection zone 3 which is irradiated with electromagnetic radiation 
from radiation sources 5 (for instance halogen lamps when sensing is performed in 
the visible light range). Conveyor 2 may be a belt conveyor, a slide, a vibrating 
pan conveyor, a free fall trajectory, or any other means for conveying materials. 
The material objects comprising the mixture of materials 1 may be conveyed 
singly or in plurality. 

A sensor array 4 (for example, a Sony Series 9000 CCD video camera 
when radiation from sources 5 is in the visible light range) is positioned to view 
the inspection zone 3 so to provide data arrays corresponding to measurements of 
electromagnetic radiation emanating from inspection zone 3 and from any 
materials 1 being conveyed through the inspection zone. The electromagnetic 
radiation emanating from materials 1 may be reflected radiation, radiation 
transmitted through materials 1 , radiation emitted from materials 1 through 
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fluorescence, or any other forms of radiation resulting fro^ranteraction of 
materials 1 with the incident radiation from sources 5. Sensor array 4 is shown 
positioned above conveyor 2 although in practice it may be located at any position 
required to give the desired view. Sensor array 4 is selected to be sensitive in the 
wavelength range of electromagnetic radiation emanating from material objects 
within the inspection zone 3 when irradiated by sources 5 which may be a single 
source or multiple sources. The geometry of sensor array 4 is determined by the 
application. For instance the sensor array 4 may be a linear array of sensors or an 
area array of sensors. It may physically span the full width and/or length of the 
inspection zone 3 or it may be more compact and use optics to scan the width 
and/or length of the inspection zone such as with a CCD camera. Sensor array 4 
may be positioned on the same side of the inspection zone 3 as is irradiation 
sources 5 or it may be positioned on the opposite side of inspection zone 3 from 
sources 5 or positioned at any other location with respect to irradiation sources 5 
and inspection zone 3. The effective wavelength ranges of sensor array 4 and 
sources 5 may each be in any one of the microwave wavelength range, the 
ultraviolet wavelength range, the visible light wavelength range, the infrared 
wavelength range, the x-ray wavelength range, or the gamma ray wavelength 
range or any combination thereof. Sensor array 4 may be fitted with special filters 
which allow only certain wavelengths of electromagnetic radiation to reach sensor 
array 4 for measurement. 

Sensor array 4 data corresponding to electromagnetic radiation 
measurements emanating from the inspection zone 3 and from material objects 1 
within the inspection zone are transmitted from sensor array 4 to computer 7 over 
transmitting cables 6 or by wireless transmission. Control of sensor array 4 
operation may also be provided by computer 7 over transmitting cables 6. Sensor 
array 4 data received by computer 7 are processed for analog to digital conversion 
if not already digital by nature of sensor array 4 and microcomputer processed into 
electronic images which are transmitted over cables 8 to touch sensitive screen 9 
which is capable of electronically registering the coordinates on the screen of a 
manual touch by a human operator 26. Touch screen 9 displays the images 
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corresponding to the sen^^data from sensor array 4 T)T tlie inspection zone' 3* and 
the materials to be sorted within the inspection zone 3. 

Human operator 26 views the images on touch screen 9 and manually 
touches the image of any material object, in this case object 25, which the 
operator wishes to be removed from the stream of materials 1 . Spatial coordinates 
describing the location on the touch screen 9 of the touch by operator 26 are 
registered by touch screen 9 and transmitted over cables 8 to computer 7. 
Computer 7 associates the touch screen coordinates of the registered touch with 
corresponding spatial location coordinates on the surface of conveyor 2 within 
inspection zone 3 and further associates any object on conveyor 2 at that location 
(in this case object 25) with the touch. Computer 7 then electronically tracks the 
motion of selected object 25 as it is further conveyed along conveyor 2 utilizing 
signals transmitted over cables 28 indicating conveyor speed generated by 
conveyor speed encoder 27. Similarly, the motion could be tracked by 
programming into computer 7 a predetermined speed for conveyor 2 in which case 
conveyor speed encoder 27 would not be needed. 

As the selected object 25 is conveyed into the vicinity of robotic arms 18 
the computer 7 performs scheduling algorithms to determine which of the robotic 
arms 18 will be used to most efficiently extract the object 25 from the waste 
stream. Such scheduling algorithms are specific to the equipment used and 
materials to be sorted and can be suitably devised by one skilled in the art using, 
for example, the techniques discussed in "Schedule Efficiency in a Robotic Cell" 
by Irina Ioachim and Francois Soumis, International Journal of Flexible 
Manufacturing Systems . Vol. 7, No. 1 (March 1995). The entire contents of this 
publication is incorporated herein by reference. There may be only one robotic 
arm 18 or as many robotic arms 18 as required to efficiently sort materials as 
selected by operator 26 on touch screen 9. For instance if the average acquisition 
and retrieval time of a robotic arm 18 for the materials to be sorted 1 is two 
seconds and the operator 26 can make an average of 2.5 selections per second then 
a minimum of five robotic arms 18 would be required to keep up with operator 26 
selections. 
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As the selected object 25 approaches, the robotic a?Brl8 chosen by the 
computer to make the extraction is commanded by computer 7 over cables 10 to 
rotate (shown as motion 24) into proper position for most efficient timing to make 
an interception of object 25. Vertical actuator 19 is positioned by computer 7 
through control signals over cables 10 to intercept object 25 as it passes under 
robotic arm 18. Vertical actuator 19 is placed by computer 7 in retracted (lifted) 
position so the object 25 can pass under it. As object 25 passes under actuator 19 
computer 7 signals the actuator via cables 10 to extend downward until end 
effector 20 (or 21) contacts object 25. Upon contact with object 25 end effector 
20 (or 21) acquires object 25 at which time actuator 19 retracts (lifts) and extracts 
object 25 from the mixture of objects 1. When object 25 is sufficiently lifted so 
that its lower extent is higher than the top of the other objects 1 the computer 7 
signals actuator 19 to translate along robotic arm 18 in one of the directions 
toward the edge of the conveyor belt 2 as indicted by directional arrows 22. 
Positioned along each side of conveyor 2 are receiver chutes 29 and 30. Actuator 
19 holding object 25 via its end effector 20 (or 21) continues moving until it is 
above either chute 29 or chute 30 at which time end effector 20 (or 21) releases 
object 25 into chute 29 or chute 30 which segregates object 25 from the other 
objects 1 on the conveyor 2 therefore having sorted object 25 from the other 
objects 1. Actuator 19 then translates back to a position over the conveyor 2 and 
awaits commands from computer 7 to make another object acquisition and retrieval 
in response to touches by operator 26 on the touch screen 9. 

Computer 7 may contain identification and pattern recognition algorithms 
which can perform learning of selections by operator 26 as the operator makes the 
selections. Such identification and pattern recognition algorithms may be 
accomplished by computerized neural networks or other such pattern recognition 
computer code, as discussed earlier herein. In this case operator 26 will initially 
make selections of items to be extracted from the mixture of materials such as 
object 25. As operator 26 makes selections the associated electronic images will 
be processed through the computer algorithms with the imaging patterns distinctive 
to the selected items noted by the algorithms. As similar items are repetitively 
selected by operator 26 the computer algorithms associate the distinctive properties 
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of the imaging patterns wn objects to be selected for extraction and begin to 
electronically select similar patterns for extraction without input from the human 
operator 26. In this way the computerized system learns those objects to be 
extracted and after sufficient learning experience can begin sorting without input 
from operator 26. 

Figures 6, 7a, and 7b show views of a robotic arm 18, a vertical actuator 
19, and an end effector 20 (or 21) which is a part of the preferred embodiment 
designed to pick up an object such as object 25 using a suction cup embodiment of 
the end effector 20. The vertical actuator 19 and end effector 20 (or 21) is also 
designed to eliminate the need for vertical actuator 19 to translate along robotic 
arm 18 after acquisition of object 25 in order to discharge object 25 into receiving 
chute 29 or receiving chute 30 (Fig. 5). The workings of this system is shown in 
Figs. 7a and 7b. When robotic arm 18 and vertical actuator 19 are positioned 
over the object 25 to be picked up, an air cylinder 32 is extended until the suction 
cup end effector 20 (or 21) contacts object 25. Vacuum is applied to the suction 
cup by vacuum hose 33 through vacuum pump 31 and through 3- way air valve 38. 
Air valve 38 at this position is set to allow the vacuum from vacuum pump 31 to 
pass through it to suction cup end effector 20 (or 21) and to not allow any 
compressed air from hose 37 to pass. Air cylinder 32 is then retracted so to lift 
object 25. As air cylinder 32 continues to retract a fixed lever 34 strikes a fixed 
extension rod 35 which causes the lower portion of the actuator 19 to rotate 
approximately 90 degrees as shown around a pivot joint 36 so that the suction cup 
end effector 20 (or 21) points outward toward the receiving chute 29 or the 
receiving chute 30 (Fig. 5). At this time the air valve 38 is actuated to close off 
vacuum to the suction cup end effector 20 (or 21) and at the same time apply 
compressed air from the hose 37 to the suction cup. The compressed air blast 39 
forcefully ejects object 25 from the suction cup end effector 20 (or 21) in a 
direction 40 and into the receiving chute 29 or the receiving chute 30. Computer 
7 then positions vertical actuator 19 for retrieval of the next object. This sequence 
of actions allows the system to pick up object 25 and eject it into receiving chutes 
29 or 30 without requiring any translation along robotic arm 18 to the receiving 
chutes. This allows a significant savings in cycle time and can reduce the number 
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of robotic arms and vertical actuators needed for the systewwnd thereby reducing 
cost. An appropriate robotic arm is comprised of Hauser HLE 100A Belt Driven 
Positioner coupled to Origa Rodless Cylinder Model 40-J2220/20x32.75-B-M 
fitted with suction cup end effector PIAB L5 Vacuum Pump with B20 Suction 
Cups. 

Figures 8a and 8b show another embodiment of a vertical actuator 19 and 
an end effector 20 (or 21) which is designed to retrieve paper sheets or cardboard 
sheets 44. In this embodiment end effector 20 (or 21) is comprised of heavy duty 
spikes 43 which can be driven or extracted by air cylinders 42. Air for control of 
air cylinders 42 is supplied through air hoses 41. In operation the vertical actuator 
19 is extended downward until the end effector 20 (or 21) contacts the paper sheet 
or cardboard 44 at which time the air cylinders 42 are actuated to rapidly and 
forcefully extend spikes 43 so that they penetrate through sheet 44 as shown. 
Vertical actuator 19 is then retracted (lifted) to lift sheet 44. Sheet 44 cannot slide 
off spikes 43 by virtue of their angled geometry. Actuator 19 is then translated to 
receiving chute 29 or 30 as previously described. When actuator 19 with sheet 44 
are positioned over the receivmg chute the spikes 43 are retracted by action of the 
air cylinders 42 and thereby withdrawing from sheet 44. This frees sheet 44 so 
that it can fall into the receiving chute. Vertical actuator 19 is then repositioned 
by computer 7 over conveyor 2 to await the next paper sheet or cardboard sheet to 
be retrieved. End effectors similar in design but much lighter in construction and 
which are designed for textile handling can be acquired from Techno Sommer, 
which has an Automatic SCH20 Needle Gripper product. 



implementing the preferred embodiment of the invention depicted in Fig. 5. In 
Fig. 11 the sensor array 4 sends data through A/D converter 46 to microprocessor 
47. Human operator 26 (Fig. 5) touches 48 the touch screen 9 on the displayed 
image to select object 25 (Fig. 5). This input to touch screen 9 is tagged 49 by 
microprocessor 47 as a selection icon 51 which is scrolled 50 in microprocessor 47 
host memory and on touch screen 9 as corresponding selected object 25 moves 
through inspection zone 3 (Fig. 5). Microprocessor 47 also enters a representation 
of selection icon 51 into the pick queue 59, assigns to the representation of 



Figure 1 1 shows a block diagram of software interfaces for 
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selection icon 51 approprfite values of position coordinates 60 to establish proper 
timing for robotic acquisition (picking) of selected object 25 (Fig. 5) and a 
scheduling assignment 61 to an appropriate robotic arm 18 (Fig. 5) to effect 
acquisition of selected object 25 as it passes under the appropriate robotic arm 18 
and deliver selected object 25 to an appropriate receiving chute 29 or 30 (Fig. 5). 
Appendix F of the DOE Report discloses sample prototype code for implementing 
an embodiment of the tagging 49 of a material object by the microprocessor 47 as 
a selection icon 51 and the subsequent tracking and generation of an ejection signal 
for the tagged item. 

Fig. 12 shows a block diagram of hardware implementation for the 
preferred embodiment of Figs. 5 and 11. In this case video camera 56 is used as 
sensor array 4 (Figs. 5 and 11) and interfaces to computer 7 (which incorporates 
microprocessor 47) through a frame grabber card 57. Images from video camera 
56 are displayed as digitized or analog electronic images on touch screen 9 (Fig. 
11). User touch input 48 (Fig. 11) is sent back to computer 7 and processed in 
conjunction with the electronic images to extract selected object 25 from a mixture 
of materials 1 (Fig. 5) by activating robotic arms 62 at the appropriate time 
through motion controller card(s) 63. An appropriate CCD video camera for this 
use is a Sony Model DCX-9000 3CCD color progressive scan camera. A 
representative video card for interfacing with the Sony camera is Coreco Model 
Ultra II with RGB Digitization Module. A representative computer is Carlo 
Gavazzi Model 690 Industrial Computer. An appropriate touch screen is PCS 
Computers Inc. Series G Flat Panel Monitor. An appropriate motion controller 
card is Parker Compumotor Motion Controller 806450. An appropriate robotic 
arm is comprised of Hauser HLE 100 A Belt Driven Positioner coupled to Origa 
Rodless Cylinder Model 40-J2220/20x32.75-B-M fitted with suction cup end 
effector PIAB L5 Vacuum Pump with B20 Suction Cups. 

Other embodiments of the invention will be apparent to those skilled in the 
art from consideration of the specification and practice of the invention disclosed 
herein. It is intended that the specification be considered as exemplary only, with 
the true scope and spirit of the invention being indicated by the following claims. 
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What is claimed is: 

1 . A method for identifying and sorting selected material objects from a 
mixture of material objects comprising steps of: 

(a) conveying a mixture of material objects into and through an 
inspection zone; 

(b) irradiating said mixture of material objects with incident 
electromagnetic radiation while in said inspection zone; 

(c) measuring the electromagnetic radiation emanating from said 
irradiated mixture of material objects; 

(d) processing said measured electromagnetic radiation to produce 
electronic images of said irradiated material objects and presenting 
said electronic images for visual display; 

(e) interactively selecting, from said visual display of material objects, 
selected material objects to be sorted from said mixture of material 
objects; and 

(f) using an automated device to separate said selected material objects 
from said mixture of material objects. 

2. The method according to claim 1 wherein said interactive selecting of 
material objects to be sorted is performed by a human operator using a 
computerized pointing device. 

3. The method according to claim 1 wherein said interactive selecting of 
material objects to be sorted is performed by a human operator and is analyzed by 
computerized learning and control algorithms so that decisions for the selecting of 
material items to be sorted is passed from the human operator to the computerized 
algorithms. 

4. The method according to claim 2 wherein said interactive selecting of 
material objects to be sorted by a human operator is analyzed by computerized 
learning and control algorithms so that identifications and selections for the said 

18 

SUBSTITUTE SHEET (RULE 26) 



BNSDOCID: <WO 9819799A1 I > 




WO 98/19799 PCT/US97/19680 



selecting of material items to be sorted is passed from the human operator to the 
computerized algorithms. 

5. The method according to claim 1 wherein said mixture of material objects 
is comprised of solid waste materials. 



6. The method according to claim 2 wherein said mixture of material objects 
is comprised of solid waste materials. 

7. The method according to claim 3 wherein said mixture of material objects 
is comprised of solid waste materials. 

8. The method according to claim 4 wherein said mixture of material objects 
is comprised of solid waste materials. 

9. The method according to claim 1 wherein said incident electromagnetic 
radiation is in the microwave wavelength range. 

10. The method according to claim 1 wherein said incident electromagnetic 
radiation is in the ultraviolet wavelength range. 

11. The method according to claim 1 wherein said incident electromagnetic 
radiation is in the visible light wavelength range. 

12. The method according to claim 1 wherein said incident electromagnetic 
radiation is in the infrared wavelength range. 

13. The method according to claim 1 wherein said incident electromagnetic 
radiation is in the x-ray wavelength range. 

14. The method according to claim 1 wherein said incident electromagnetic 
radiation is in the gamma ray wavelength range. 
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The method according to claim 2 wherein said incident electromagnetic 
radiation is in the microwave wavelength range. 

The method according to claim 2 wherein said incident electromagnetic 
radiation is in the ultraviolet wavelength range. 

The method according to claim 2 wherein said incident electromagnetic 
radiation is in the visible light wavelength range. 

The method according to claim 2 wherein said incident electromagnetic 
radiation is in the infrared wavelength range. 

The method according to claim 2 wherein said incident electromagnetic 
radiation is in the x-ray wavelength range. 

The method according to claim 2 wherein said incident electromagnetic 
radiation is in the gamma ray wavelength range. 

The method according to claim 3 wherein said incident electromagnetic 
radiation is in the microwave wavelength range. 

The method according to claim 3 wherein said incident electromagnetic 
radiation is in the ultraviolet wavelength range. 

The method according to claim 3 wherein said incident electromagnetic 
radiation is in the visible light wavelength range. 

The method according to claim 3 wherein said incident electromagnetic 
radiation is in the infrared wavelength range. 

The method according to claim 3 wherein said incident electromagnetic 
radiation is in the x-ray wavelength range. 
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26. The method according to claim 3 wherein said incident electromagnetic 
radiation is in the gamma ray wavelength range. 



27. The method according to claim 4 wherein said incident electromagnetic 
radiation is in the microwave wavelength range. 

28. The method according to claim 4 wherein said incident electromagnetic 
radiation is in the ultraviolet wavelength range. 

29. The method according to claim 4 wherein said incident electromagnetic 
radiation is in the visible light wavelength range. 

30. The method according to claim 4 wherein said incident electromagnetic 
radiation is in the infrared wavelength range. 

31. The method according to claim 4 wherein said incident electromagnetic 
radiation is in the x-ray wavelength range. 

32. The method according to claim 4 wherein said incident electromagnetic 
radiation is in the gamma ray wavelength range. 

33. An apparatus for identifying and sorting selected material objects from a 
mixture of material objects comprising: 

(a) a conveyor that conveys a mixture of material objects into and 
through an inspection zone; 

(b) an electromagnetic radiation source that irradiates said mixture of 
material objects within said inspection zone with incident 
electromagnetic radiation; 

(c) a sensor that examines said material objects within said inspection 
zone and measures electromagnetic radiation emanating from said 
material objects; 

(d) a microprocessor that processes said measured electromagnetic 
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radiation to produce electronic images of said material objects as 
they pass through said inspection zone and a display coupled to said 
microprocessor that presents a visual display of said electronic 
images; and 

(e) a human operator interface to said visual display such that an 
operator may select selected material objects to be sorted from said 
mixture of objects; and 

(f) an automated device that separates said selected material objects 
from said mixture of material objects. 

34. The apparatus according to claim 33 wherein said mixture of material 
objects is comprised of solid waste materials. 

35. The apparatus according to claim 33 wherein said sensor includes a CCD 
camera. 

36. The apparatus according to claim 33 wherein said conveyor is taken from a 
group consisting of a belt conveyor, a vibrating pan conveyor, a slide, and 
a free fall trajectory. 

37. The apparatus according to claim 33 wherein said electromagnetic radiation 
source includes a lamp emitting radiation in the visible range. 

38 The apparatus according to claim 33 wherein said display includes a 
computer monitor. 

39. The apparatus according to claim 33 wherein said human operator interface 
includes a touch sensitive screen. 

40. The apparatus according to claim 33 wherein said automated device 
includes an air ejector. 
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41. The apparatus according to claim 33 wherein said automated device 
includes a robotic arm. 

42. The apparatus according to claim 41 wherein said robotic arm includes a 
suction cup end effector. 

43. The apparatus according to claim 41 wherein said robotic arm includes a 
driven and retractable spike end effector. 

44. The apparatus according to claim 42 wherein said suction cup includes 
means to rotate said suction cup upward and which further includes means 
for disconnecting vacuum to said suction cup and means to provide 
compressed air to said suction cup to propel an object held by said suction 
cup into a receiving chute or bin. 

45. The apparatus according to claim 33 wherein said electromagnetic radiation 
emanating from said material objects as they pass through said inspection 
zone is passed through a filter to pass only selected wavelengths prior to 
measuring said electromagnetic radiation. 

46 The apparatus according to claim 33 wherein said electromagnetic radiation 
source provides incident electromagnetic radiation in the microwave 
wavelength range. 

47. The apparatus according to claim 33 wherein said electromagnetic radiation 
source provides incident electromagnetic radiation in the ultraviolet 
wavelength range. 

48. The apparatus according to claim 33 wherein said electromagnetic radiation 
source provides incident electromagnetic radiation in the visible wavelength range. 
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49. The apparatus according to claim 33 wherein said electromagnetic radiation 
source provides incident electromagnetic radiation in the infrared 
wavelength range. 

50. The apparatus according to claim 33 wherein said electromagnetic radiation 
source provides incident electromagnetic radiation in the x-ray wavelength 
range. 

51 The apparatus according to claim 33 wherein said electromagnetic radiation 
source provides incident electromagnetic radiation in the gamma ray 
wavelength range. 
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(57) Abstract 



A method and apparatus are disclosed 
for classifying materials (16) utilizing a 
computerized touch sensitive screen (9) or 
other computerized pointing device for op- 
erator identification and electronic marking 
of spatial coordinates of materials (16) to be 
extracted. An operator positioned at a com- 
puterized touch sensitive screen (9) views 
electronic images of the mixture of materi- 
als (16) to be sorted as they are conveyed 
past a sensor array (4) which transmits se- 
quences of images of the mixture, either di- 
rectly or through a computer to the touch 
sensitive display screen (9). The operator 
manually "touches" objects displayed on the 
screen (9) to be extracted from the mixture 
thereby registering the spatial coordinates 
of the objects within the computer. The 
computer then tracks the registered objects 
as they are conveyed and directs automated 
devices including mechanical means such as 
air jets (15), robotic arms (18), or other me- 
chanical diverters to extract the registered 
objects. 
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